[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target by lym953 · Pull Request #1413 · DataDog/datadog-lambda-extension

lym953 · 2026-10-07T16:44:45Z

Stacked on #1412. Review that PR first. The new variable is parsed by the crate datadog-serverless-logging, from DataDog/serverless-components#180.

Problem

DD_LOG_LEVEL=debug turns on debug logs from all parts of the extension. In a test, three cold starts at debug level logged 117 debug lines, when only 33 of them were cold start durations. Too many logs can be noisy for users (either customers or Datadog engineers) who just want to debug cold starts.

This PR

This PR adds DD_LOG_LEVEL_BY_TARGET, a comma-separated list of <target>=<level> entries. Each entry sets the log level of one tracing target, and DD_LOG_LEVEL stays the level of all other targets. Both are environment variables of the Lambda function. For example:

Environment variable	Value
`DD_LOG_LEVEL`	`info`
`DD_LOG_LEVEL_BY_TARGET`	`cold_start_duration=debug,dogstatsd=debug`

Suppose there are logging statements with target cold_start_duration, like:

debug!(
    target: "cold_start_duration",
    "Datadog Next-Gen Extension ready in {:}ms",
    start_time.elapsed().as_millis().to_string()
);

Then here are some examples (each row is one scenario):

`DD_LOG_LEVEL`	`DD_LOG_LEVEL_BY_TARGET`	Logs that you get
`info`	`cold_start_duration=debug`	Info logs, plus the cold start step durations from #1412
`debug`	`cold_start_duration=off`	All debug logs except the cold start step durations
`info`	`dogstatsd=debug`	Info logs, plus debug logs from the `dogstatsd` crate of `serverless-components`
not set	`cold_start_duration=debug`	Info logs, plus the cold start step durations. Info is the default level.

A target can be a custom target that the logging statement sets, a module path such as bottlecap::traces, or a crate name such as dogstatsd. An entry matches every target that starts with its text. The README of datadog-serverless-logging lists all the kinds of targets.

The extension logs a warning for an entry that is not valid, and skips it. Examples are a level with no target such as debug, which would otherwise replace DD_LOG_LEVEL, and an unknown level such as cold_start_duration=verbose.

Why not `DD_LOG_LEVEL`

The Datadog libraries and tracers in the function also read DD_LOG_LEVEL, and they accept only a plain level. For example, datadog-lambda-python logs "Invalid log level" for any other value, and dd-trace-rs logs a parse error. Support for <target>=<level> entries in DD_LOG_LEVEL would need changes in several repositories. These include tracers such as dd-trace-java and dd-trace-dotnet, which Serverless teams do not own and which run in many environments other than Lambda.

Testing

I built an arm64 layer of this branch at 4a71d023 and deployed it on a Python 3.13 arm64 function. Since then, the branch was rebased onto main, changed one code comment, and moved to the merge commit of #180, which has the same code. For each row, I forced 2 cold starts and counted the extension's log lines in CloudWatch. Each cold start logs 11 cold start duration lines.

`DD_LOG_LEVEL`	`DD_LOG_LEVEL_BY_TARGET`	Debug lines	Cold start duration lines	Warnings
`info`	`cold_start_duration=debug`	22	22	0
`debug`	`cold_start_duration=off`	56	0	0
`info`	`debug,cold_start_duration=verbose,cold_start_duration=debug`	22	22	2
`info`	`dogstatsd=debug`	10	0	0
not set	`cold_start_duration=debug`	22	22	0

In the third row, the extension ignored debug and cold_start_duration=verbose with a warning, and debug did not turn on the other debug logs. In the fourth row, all the debug messages come from the dogstatsd crate. I did not test on amd64 or FIPS builds.

🤖 Generated with Claude Code

lym953 · 2026-10-07T19:28:14Z

Changing back to draft. Will make the change in serverless-components and make DD_LOG_LEVEL support values like: warn,cold_start_duration=debug,trace_flush_duration=debug.

lym953 · 2026-10-08T19:00:51Z

Publishing this again since the serverless-components PR is published: DataDog/serverless-components#178

Copilot

🟡 Changes recommended

The bundled wrapper does not support directive lists, and the dependencies remain pinned to a temporary upstream revision.

2 open findings

Replace temporary serverless-components pins with final merge commit · New
Wrapper fails to parse combined DD_LOG_LEVEL directives · New

What changed in this PR

Adds target-specific DD_LOG_LEVEL filtering while preserving default logging behavior.

Changes:

Parses comma-separated tracing filter directives and reports invalid entries.
Adds unit coverage for filtering behavior.
Updates serverless-components dependencies.

File	Description
`bottlecap/src/logger.rs`	Documents the cold-start logging target.
`bottlecap/src/bin/bottlecap/main.rs`	Implements and tests directive-based filtering.
`bottlecap/Cargo.toml`	Updates shared component revisions.
`bottlecap/Cargo.lock`	Locks the updated revisions.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟢 Approval recommended

The implementation matches the documented behavior and includes focused coverage for valid, invalid, and overriding directives.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

lym953 · 2026-10-09T17:20:14Z

Converting to draft again. Will move the helper function to a crate in serverless-components, as discussed with @duncanpharvey .

DD_LOG_DEBUG_TARGETS is a comma-separated list of tracing targets that log at debug level while DD_LOG_LEVEL stays at its normal level. For example, DD_LOG_DEBUG_TARGETS=cold_start_duration shows only the cold start step durations. Each entry is parsed on its own. An entry that does not parse, such as "cold_start_duration=trace", is logged as a warning and skipped, because the filter's expect() would otherwise fail startup. At DD_LOG_LEVEL=debug or trace the variable is ignored, since a target directive could only lower trace to debug. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

DD_LOG_LEVEL_BY_TARGET takes comma-separated `target=level` entries in the RUST_LOG syntax, so that a target can also get a level other than debug, or be made quieter than DD_LOG_LEVEL. An entry with no level turns on all levels for that target. Entries also override the built-in h2, hyper, reqwest and rustls defaults. Each entry is used as written, so the rule that ignored the variable at DD_LOG_LEVEL=debug or trace no longer applies. Invalid entries are still logged as a warning and skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ther Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop DD_LOG_LEVEL_BY_TARGET and read the same `target=level` directives from DD_LOG_LEVEL, for example `info,cold_start_duration=debug`. This matches the Agent Data Plane and datadog-serverless-compat, which both accept directives in DD_LOG_LEVEL. A plain level still works as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Without it, the config crate logs "Invalid log level" at ERROR on each cold start when DD_LOG_LEVEL holds filter directives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

DD_LOG_LEVEL is also read by the Datadog libraries and tracers in the function, and they accept only a plain level, so directives there would change or break their logging. A separate variable that only the extension reads avoids that, and no longer needs the serverless-components change. Restores the code of 2afd657 and documents that targets include dependency module paths, such as `dogstatsd`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Each entry must be a target, `=`, and a level name. A level with no target, such as `debug`, used to replace DD_LOG_LEVEL, and a target with no level or a span filter such as `[span]` was also accepted. These are now invalid entries, which the extension logs as a warning and skips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The parsing and the variable name move to the new crate in serverless-components (DataDog/serverless-components#180), so that other serverless agents can share them. Points all five serverless-components dependencies at the head of #180. Its other changes since the current revision are docs and a lock file bump that the extension does not use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…stant The crate README of datadog-serverless-logging documents how to set the variable and the prefix match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The temporary upstream revision must be replaced with the final merge commit before approval.

1 open finding

Repin dependencies and regenerate Cargo.lock after upstream merge · New

🧠 Review effort: Balanced

+dogstatsd = { git = "https://github.com/DataDog/serverless-components", rev = "e5524046ed1184b32de656e3a86eed91976180f8", default-features = false }
+datadog-fips = { git = "https://github.com/DataDog/serverless-components", rev = "e5524046ed1184b32de656e3a86eed91976180f8", default-features = false }
+datadog-agent-config = { git = "https://github.com/DataDog/serverless-components", rev = "e5524046ed1184b32de656e3a86eed91976180f8", default-features = false }
+datadog-agent-trace-sampler = { git = "https://github.com/DataDog/serverless-components", rev = "e5524046ed1184b32de656e3a86eed91976180f8", default-features = false }
+datadog-serverless-logging = { git = "https://github.com/DataDog/serverless-components", rev = "e5524046ed1184b32de656e3a86eed91976180f8", default-features = false }


clifordshelton

Thanks for moving the filter logic into a shared crate. Keeping it out of DD_LOG_LEVEL makes sense too, since the tracers and libraries also read that variable. I have two inline comments and two broader ones here.

1. Decide whether DD_LOG_LEVEL_BY_TARGET is for customers or for us

This adds a new environment variable that customers can set, but it isn't documented anywhere they'd see it. The README in serverless-components#180 is written for developers. Before we document it, it's worth noticing what the setting depends on. Besides custom targets like cold_start_duration, it accepts our internal Rust module paths, like bottlecap::traces, and crate names with underscores. Those can change any time we move or rename code. If customers start using them, a refactor on our side would quietly change what their setting does, and nobody would notice.

My suggestion is to treat it as a support and debugging tool for now. Only document the custom targets we add on purpose (cold_start_duration, and later ones like trace_flush_duration), since those are names we can keep stable. Module paths still work, but we don't promote them. That way we can add or rename modules without breaking anyone. It would be good to confirm this with the team before merging, since the name and behavior are hard to change once customers rely on them.

2. Bring back the note about target names

An entry matches every target that starts with its text, so trace_flush=debug would also turn on trace_flush_duration. There was a code comment warning about this, but the "Drop the DD_LOG_LEVEL_BY_TARGET notes" commit removed it, and now it's only in the serverless-components README. Whoever adds the next target will be working in logger.rs, not reading that README. It's better to put the note back next to COLD_START_DURATION in logger.rs, so new target names don't accidentally start with an existing one.

clifordshelton · 2026-10-09T20:12:56Z

-    let env_filter = format!(
-        "h2=off,hyper=off,reqwest=off,rustls=off,datadog-trace-mini-agent=off,{log_level:?}",
-    );
+    let (env_filter, invalid_entries) = build_env_filter(


The base filter turns off h2, hyper, reqwest and rustls for a reason, and with this change a DD_LOG_LEVEL_BY_TARGET entry can turn them back on. The extension reads its own stdout and sends it to Datadog as logs (see the "Be careful adding logs inside the logs pipeline" note in AGENTS.md). These HTTP crates log on every request, and that includes the requests the extension makes to send logs. So with hyper=debug, every flush creates more log lines, which get sent in the next flush, which creates more lines, and so on. That adds a lot of noise to the customer's CloudWatch and Datadog logs, and it costs them money.

Since this is an environment variable customers can set, I'd keep those four crates locked to off. You can skip any entry for them and add it to invalid_entries, so the user gets the same warning as for a typo. The cleanest place for this is probably the crate in serverless-components#180, for example by letting the caller pass a list of targets that can't be changed. That would mean making the change before #180 merges. The overrides_base_filter test that sets h2=debug would then change to check that the entry is rejected.

The merge commit has the same tree as the branch head that this PR used before, so the code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… constant A DD_LOG_LEVEL_BY_TARGET entry matches every target that starts with its text. The next target is added in logger.rs, so the warning belongs there, not only in the serverless-components README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lym953 mentioned this pull request Oct 7, 2026

[SVLS-9883] feat: log the duration of each cold start step #1412

Open

This comment has been minimized.

Sign in to view

lym953 force-pushed the yiming.luo/log-debug-targets branch from 9862713 to 24a9990 Compare October 7, 2026 17:25

lym953 changed the title ~~[SVLS-9883] feat: add DD_LOG_DEBUG_TARGETS to turn on debug logs per target~~ [SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target Oct 7, 2026

lym953 marked this pull request as ready for review October 7, 2026 18:56

lym953 requested a review from a team as a code owner October 7, 2026 18:56

lym953 requested a review from duncanista October 7, 2026 18:56

lym953 marked this pull request as draft October 7, 2026 19:27

lym953 mentioned this pull request Oct 7, 2026

[SVLS-9883] feat(config): accept filter directives in DD_LOG_LEVEL DataDog/serverless-components#178

Closed

lym953 changed the title ~~[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target~~ [SVLS-9883] feat: accept per-target log levels in DD_LOG_LEVEL Oct 7, 2026

lym953 marked this pull request as ready for review October 8, 2026 19:00

lym953 requested review from a team and clifordshelton and a balanced review from Copilot and removed request for a team and duncanista October 8, 2026 19:01

Copilot started reviewing on behalf of lym953 October 8, 2026 19:01

View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread bottlecap/Cargo.toml Outdated

Comment thread bottlecap/src/bin/bottlecap/main.rs Outdated

lym953 marked this pull request as draft October 9, 2026 15:39

lym953 changed the title ~~[SVLS-9883] feat: accept per-target log levels in DD_LOG_LEVEL~~ [SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target Oct 9, 2026

lym953 requested a balanced review from Copilot October 9, 2026 17:06

Copilot started reviewing on behalf of lym953 October 9, 2026 17:06

View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

lym953 marked this pull request as ready for review October 9, 2026 17:08

lym953 marked this pull request as draft October 9, 2026 17:19

lym953 mentioned this pull request Oct 9, 2026

[SVLS-9883] feat: add datadog-serverless-logging crate for DD_LOG_LEVEL_BY_TARGET DataDog/serverless-components#180

Merged

lym953 and others added 10 commits October 9, 2026 14:56

[SVLS-9883] Use trace_flush_duration in the env filter test

246a2e9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[SVLS-9883] Document that target names must not be prefixes of each o…

16f683a

…ther Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[SVLS-9883] Bump serverless-components for directive-aware DD_LOG_LEVEL

0e706b3

Without it, the config crate logs "Invalid log level" at ERROR on each cold start when DD_LOG_LEVEL holds filter directives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[SVLS-9883] Bump serverless-components to the latest commit of #178

35ea498

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[SVLS-9883] Bump serverless-components to the rebased #178

015e783

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lym953 force-pushed the yiming.luo/cold-start-duration-logs branch from 776fb27 to acf98dd Compare October 9, 2026 18:56

lym953 force-pushed the yiming.luo/log-debug-targets branch from 4a71d02 to e47df3c Compare October 9, 2026 18:56

[SVLS-9883] Drop the DD_LOG_LEVEL_BY_TARGET notes from the target con…

c2aa611

…stant The crate README of datadog-serverless-logging documents how to set the variable and the prefix match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lym953 marked this pull request as ready for review October 9, 2026 19:14

lym953 requested a balanced review from Copilot October 9, 2026 19:14

Copilot started reviewing on behalf of lym953 October 9, 2026 19:14

View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

clifordshelton reviewed Oct 9, 2026

View reviewed changes

lym953 and others added 2 commits October 13, 2026 12:43

[SVLS-9883] Bump serverless-components to the merge commit of #180

f9fd99a

The merge commit has the same tree as the branch head that this PR used before, so the code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lym953 mentioned this pull request Oct 14, 2026

[SVLS-9883] feat: let agents lock targets in DD_LOG_LEVEL_BY_TARGET DataDog/serverless-components#183

Open

Navigation Menu

Repository navigation

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target - #1413

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target#1413
lym953 wants to merge 14 commits into
yiming.luo/cold-start-duration-logsfrom
yiming.luo/log-debug-targets

lym953 commented Oct 7, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

lym953 commented Oct 7, 2026

Uh oh!

lym953 commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

lym953 commented Oct 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

clifordshelton left a comment

Uh oh!

Uh oh!

clifordshelton Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lym953 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

This PR

Why not DD_LOG_LEVEL

Testing

Uh oh!

This comment has been minimized.

lym953 commented Oct 7, 2026

Uh oh!

lym953 commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

lym953 commented Oct 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

clifordshelton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

clifordshelton Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lym953 commented Oct 7, 2026 •

edited

Loading

Why not `DD_LOG_LEVEL`